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METHOD AND APPARATUS FOR USER-CONTROLLED SELECTIVE 
OVERLAY IN A STREAMING MEDIA 

CROSS REFERENCE TO RELATED APPLICATIONS 

5 

The present invention is related to applications 
entitled METHOD AND APPARATUS FOR DYNAMIC DISTRIBUTION OF 
CONTROLLED AND ADDITIONAL SELECTIVE OVERLAYS IN A 
STREAMING MEDIA, serial no. , attorney docket 

10 no. AT9-99-250; and METHOD AND APPARATUS FOR DYNAMIC 
DISTRIBUTION OF CONTROLLED AND ADDITIONAL SELECTIVE 

OVERLAYS IN A STREAMING MEDIA, serial no. , 

attorney docket no. AT9-99-255; all of which are filed 
even date hereof, assigned to the same assignee, and 

15 incorporated herein by reference. 

BACKGROUND OF THE INVENTION 

20 1. Technical Field: 

The present invention relates to the field of 
computer software and, more particularly, to methods of 
providing streaming media to users. 

25 2. Description of Related Art: 

Internet, also referred to as an "internetwork", in 
communications is a set of computer networks, possibly 
dissimilar, joined together by means of gateways that 
handle data transfer and the conversion of messages from 
30 the sending network to the protocols used by the receiving 
network (with packets if necessary) . When capitalized, 
the term "Internet" refers to the collection of networks 
and gateways that use the TCP/IP suite of protocols. 

The Internet has become a cultural fixture as a 



Docket No. AT9-99-254 

source of both information and entertainment. Many 
businesses are creating Internet sites as an integral part 
of their marketing efforts^ informing consumers of the 
products or services offered by the business or providing 
5 other information seeking to engender brand loyalty. Many 
federal, state, and local government agencies are also 
employing Internet sites for informational purposes, 
particularly agencies which must interact with virtually 
all segments of society such as the Internal Revenue 

10 Service and secretaries of state. Operating costs may be 
reduced by providing informational guides and/or 
searchable databases of public records online. 

Furthermore, the Internet has destroyed traditional 
geographic and technical barriers to distribution of 

15 multimedia content. Traditionally, multimedia content has 
been distributed via broadcast television (sometimes in 
combination with radio), moving pictures (film), and, more 
recently, cable television. Much of this multimedia 
content was only available within limited geographic areas 

20 and presented in one language. 

To reach wider audiences and provide consumers with 
more choices, television broadcasts have employed SAP 
broadcasting in which a television broadcast contained 
audio tracks for several different languages for the same 

25 video content. However, SAP is an all or nothing choice. 
That is, a user must choose one audio track. The 
possibility of mixing several audio tracks together is 
non-existent using SAP. 

Another method television broadcasters have used to 

30 provide users with more choices over the presentation of 
prograimning is "'simulcast''. With this technology, a 
different audio track was broadcast via radio to 
correspond to a simultaneously broadcast television 
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program. However^ this involved the use of two different 
media. Furthermore, only the whole of the audio sound 
track was broadcast, not pieces of it which could be 
selectively filtered. Additionally, this method did not 
5 contain any other video components and there was not a 
single point of control. ^'Simulcast" required two points 
of control: a television and a separate radio. 

Currently, the potential is available for anyone 
anywhere in the world to view content being broadcast from 

10 anywhere in the world. This potential needs to be 

realized and requires a new technology to reduce overall 
bandwidth while providing viewer centric content as a way 
of differentiating a broadcaster from his competition in 
the world market. Also, to encourage viewers to watch 

15 an event or program it is desirable to appeal to as many 
different types of viewers as possible. In some cases, 
events or programs may be broadcast in different 
languages. This type of customization typically requires 
a separate broadcast for each language. This type of 

20 customization requires bandwidth for the audio and video 
for each version. As a result, bandwidth is wasted by 
rebroadcasting an event for each versions. Thus, it 
would be advantageous to have an improved method and 
apparatus to allow customization of the presentation of 

25 events to fit the viewing patterns, wants, and needs of 
present day consumers and reduce bandwidth needed to 
customize events. 
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SUMMARY OF THE INVENTION 

The present invention provides a method and apparatus for 
tailoring a multimedia presentation of an event on a 

5 computerized multimedia system, A set of video streams^ a 
set of audio streams and a set of information streams are 
provided for the event via a network coupled to the 
computerized multimedia system. Video streams are 
selected for presentation from the set of video streams 

10 for the event. Audio streams are selected for 

presentation from the set of audio streams for the event. 
Information streams are selected for presentation from 
the set of information streams for the event. Responsive 
to user input;, selected video streams and the selected 

15 audio streams are assigned to respective portions of 

video and audio output devices. The event is presented 
according to selected video stream and audio stream 
assignments . 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The novel features believed characteristic of the 
5 invention are set forth in the appended claims. The 

invention itself^ however, as well as a preferred mode of 
use, further objectives and advantages thereof, will best 
be understood by reference to the following detailed 
description of an illustrative embodiment when read in 
10 conjunction with the accompanying drawings, wherein: 
Figure 1 depicts a pictorial representation of a 
distributed data processing system in which the present 
invention may be implemented; 

Figure 2 depicts a block diagram of a data 
15 processing system which may be implemented as a server in 
accordance with the present inventions- 
Figure 3 depicts a block diagram of a data 
processing system in which the present invention may be 
implemented; 

20 Figure 4 shows a block diagram broadly illustrating 

the process of the present invention; 

Figure 5 is a block diagram of functional components 
used in the selection of media streams depicted in 
accordance with the present invention; 
25 Figure 6 is a diagram illustrating an example 

display depicted in accordance with a preferred 
embodiment of the present invention; 

Figures 7A-7D are diagrams illustrating data 
structures contained within data streams sent to a client 
30 depicted in accordance to a preferred embodiment of the 
present invention; 

Figure 8 is a flowchart of a process used to process 
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data streams depicted in accordance with a preferred 

embodiment of the present invention; 

Figure 9 is a flowchart of a process for generating 

CRC values for use in synchronizing data streams depicted 
5 in accordance with a preferred embodiment of the present 

invention; and 

Figure 10 is a flowchart of a process for 

synchronizing data streams using CRC data depicted in 

accordance with a preferred embodiment of the present 
10 invention. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

With reference now to the figures, and in particular 

5 with reference to Figure 1, a pictorial representation of 
a distributed data processing system is depicted in which 
the present invention may be implemented. 

Distributed data processing system 100 contains 
network 102, which is the medium used to provide 

10 communications links between various devices and 

computers connected within distributed data processing 
system 100. Network 102 may include permanent 
connections, such as wire or fiber optic cables, or 
temporary connections made through telephone connections. 

15 In the depicted example, server system 104 is 

connected to network 102, along with storage unit 106. 
Server system 104 may contain one or more server 
computers for providing data streaming. For example, a 
server may be used to administer and control data streams 

20 while another server within server system 104 is employed 
to originate resource data streams to clients. The 
information in these data streams may be stored within 
server system 104 or on a remote storage device, such as 
storage device 106. In addition, clients 108, 110 and 

25 112 are also connected to network 102. These clients, 
108, 110 and 112, may be, for example, without 
limitation, personal computers, set top boxes, web TV 
units, or any other hardware unit for receiving data 
streams. Clients 108, 110 and 112 are clients to server 

30 system 104. In the depicted example, server system 104 
provides data streams to clients 108-112. Distributed 
data processing system 100 may include additional 
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servers, clients, and other devices not shown. 

Clients 108, 110, and 112 include mechanisms for 
mixing and filtering data streams received from server 
system 104 or other server systems not shown. These 
5 mechanisms may be implemented in hardware, software, 
firmware, or some combination thereof in the depicted 
examples . 

Distributed data processing system 100 may be, for 
example, the Internet, with network 102 representing a 

10 worldwide collection of networks and gateways that use 
the TCP/IP suite of protocols to communicate with one 
another. At the heart of the Internet is a backbone of 
high-speed data communication lines between major nodes 
or host computers consisting of thousands of commercial, 

15 government, education, and other computer systems that 
route data and messages. 

Another implementation for network 102 is as a 
broadband network, also referred to as a broadband 
integrated services digital network (BISDN) . In such a 

20 network, broadband services typically provide channels 
capable of supporting data transmission rates of greater 
than 1.5Mbps or a primary rate in integrated services 
digital network (ISDN), Tl, or DSl. In broadband system, 
the type of transfer mode typically is a synchronous 

25 transfer mode (ATM) with the use of synchronous optical 
network (SONET) and intelligent network (IN) technologies 
as providing the framework for ISDN. In such a network, 
information including voice, data, video, and audio are 
transferred through the network to various clients. 

30 Further, network 102 also may be a cable 

communications system depending on the implementation. Of 
course, distributed data processing system 100 also may 



9 

Docket No- AT9-99-254 



be implemented as a number of different types of networks 
such as, for example, an intranet or a local area 
network. Figure 1 is intended as an example and not as 
an architectural limitation for the processes of the 

5 present invention . 

Referring to Figure 2, a block diagram of a data 
processing system, which may be implemented as a server 
system, such as server system 104 in Figure 1, is 
depicted in accordance with the present invention. Data 

10 processing system 200 may be employed as a video server 
or other application server within a server system. 

Data processing system 200 may be a symmetric 
multiprocessor (SMP) system including a plurality of 
processors 202 and 204 connected to system bus 206. 

15 Alternatively, a single processor system may be employed. 
Also connected to system bus 206 is memory 
controller/cache 208, which provides an interface to 
local memory 209. I/O bus bridge 210 is connected to 
system bus 206 and provides an interface to I/O bus 212. 

20 Memory controller/cache 208 and I/O bus bridge 210 may be 
integrated as depicted. 

Peripheral component interconnect (PCI) bus bridge 
214 connected to I/O bus 212 provides an interface to PCI 
local bus 216. A number of modems 218-220 may be 

25 connected to PCI bus 216. Typical PCI bus 

implementations will support four PCI expansion slots or 
add-in connectors. Communications links to network 
computers 108-112 in Figure 1 may be provided through 
modem 218 and network adapter 220 connected to PCI local 

30 bus 216 through add-in boards. 

Additional PCI bus bridges 222 and 224 provide 
interfaces for additional PCI buses 226 and 228, from 
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which additional modems or network adapters may be 
supported. In this manner, server 200 allows connections 
to multiple network computers. A memory mapped graphics 
adapter 230 and hard disk 232 may also be connected to 
I/O bus 212 as depicted, either directly or indirectly- 

Those of ordinary skill in the art will appreciate 
that the hardware depicted in Figure 2 may vary. For 
example, other peripheral devices, such as optical disk 
drives and the like, also may be used in addition to or 
in place of the hardware depicted. The depicted example 
is not meant to imply architectural limitations with 
respect to the present invention. 

The data processing system depicted in Figure 2 may 
be, for example, an IBM RS/6000, a product of 
International Business Machines Corporation in Armonk, 
New York, running the Advanced Interactive Executive 
(AIX) operating system. 

With reference now to Figure 3, a block diagram of a 
data processing system in which the present invention may 
be implemented is illustrated. Data processing system 
300 is an example of a computer. Data processing system 
300 employs a peripheral component interconnect (PCI) 
local bus architecture. Although the depicted example 
employs a PCI bus, other bus architectures, such as Micro 
Channel and ISA, may be used. Processor 302 and main 
memory 304 are connected to PCI local bus 306 through PCI 
bridge 308. PCI bridge 308 may also include an 
integrated memory controller and cache memory for 
processor 302. Additional connections to PCI local bus 
306 may be made through direct component interconnection 
or through add-in boards. In the depicted example, local 
area network (LAN) adapter 310, SCSI host bus adapter 
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312, and expansion bus interface 314 are connected to PCI 
local bus 306 by direct component connection. In 
contrast, audio adapter 316, graphics adapter 318, and 
audio/video adapter (A/V) 319 are connected to PCI local 
5 bus 306 by add-in boards inserted into expansion slots. 
Expansion bus interface 314 provides a connection for a 
keyboard and mouse adapter 320, modem 322, and additional 
memory 324. In the depicted example, SCSI host bus 
adapter 312 provides a connection for hard disk drive 

10 326, tape drive 328, CD-ROM drive 330, and digital video 
disc read only memory drive (DVD-ROM) 332. Typical PCI 
local bus implementations will support three or four PCI 
expansion slots or add-in connectors. 

An operating system runs on processor 302 and is 

15 used to coordinate and provide control of various 

components within data processing system 300 in Figure 3. 
The operating system may be a commercially available 
operating system, such as OS/2, which is available from 
International Business Machines Corporation, ''OS/2" is a 

20 trademark of International Business Machines Corporation. 
An object oriented programming system, such as Java, may 
run in conjunction with the operating system, providing 
calls to the operating system from Java programs or 
applications executing on data processing system 300. 

25 Instructions for the operating system, the 

object-oriented operating system, and applications or 
programs are located on a storage device, such as hard 
disk drive 326, and may be loaded into main memory 304 
for execution by processor 302. 

30 Those of ordinary skill in the art will appreciate 

that the hardware in Figure 3 may vary depending on the 
implementation. For example, other peripheral devices. 
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such as optical disk drives and the like, may be used in 
addition to or in place of the hardware depicted in 
Figure 3. The depicted example is not meant to imply 
architectural limitations with respect to the present 
5 invention. For example, the processes of the present 
invention may be applied to multiprocessor data 
processing systems. Additionally, data processing system 
300 may take the form of a notebook computer or a 
personal digital assistant (PDA) . Further, the data 

10 processing system may be, for example, without 

limitations, a television or settop cable box containing 
appropriate hardware and software for implementing 
processes of the present invention. 

The present invention provides a method, apparatus, 

15 and computer implemented instructions for decreasing 
bandwidth usage and for providing increased client 
customization of programs or events. The mechanism of 
the present invention is directed towards tailoring a 
multimedia presentation on a multimedia data processing 

20 system. This is accomplished by breaking up a program or 
event into different data streams for different audio and 
video components. Additional optional data streams may 
be added to allow for customization of the program or 
event. A data processing system may receive a set of 

25 video and audio streams associated with a program or 

event. These data streams may be selectively presented 
based on user input. Depending on the user input, some 
portions of the video and audio data streams may be 
presented while others are omitted. Further, user input 

30 may selectively place a video stream on a display. Also, 
information streams containing text and/or graphics may 
be received and selectively displayed in association with 
the presentation of the program or event. The 
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combination of these data streams at the user site 
provides for customization without requiring broadcasting 
of multiple versions of an event to viewers. 

By allowing a viewer to customize a program or 
5 eventy. bandwidth usage may be reduced because program 

originators do not have to broadcast multiple versions of 
a program or event for different geographical regions or 
different viewer basis. 

Turning now to Figure A, there is shown a block 

10 diagram broadly illustrating the process of the present 
invention. A client 400, such as data processing system 
300 in Figure 3, sends a request 402 for programming 
content to an originating server 404. Originating server 
404 is a data processing system, such as data processing 

15 system 200 in Figure 2, and is the host for the 

programming content requested by client 400. Originating 
server 404 sends the requested programming as a set of 
data streams 406 back to client 400. Data streams 406, 
may be used to present programming or events to a user at 

20 client 402. Furthermore, data streams 406 may comprise 
one or more video stream components, one or more 
information stream components, and one or more audio 
stream components. 

Client 400 and originating server 404 have been 

25 described and illustrated in Figure 4 as though they are 
directly connected. However, client 402 and originating 
server 404 are preferably part of a distributed data 
processing system such as distributed data processing 
system 100 and are linked together via a network, such as 

30 network 102 in Figure 1. 

Turning next to Figure 5, a block diagram of 
functional components used in the selection of media 
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streams in accordance with the present invention is 
depicted. In this example, client 500 includes a data 
stream processing system 502 to receive data streams 510 
from a network 504 for output to output system 506 • The 
5 selection of data streams received from network 504 are 
controlled through user control 508^ which controls data 
stream processing system 502. 

Multiple data streams may be received from network 
504 through connection 508. These data streams may be, 

10 for example, audio, video, text, graphics, or other 

information that may be displayed through output system 
506 to a user of client 500. Selection filtering of data 
streams from network 504 by data stream processing system 
502 are controlled through user control 508. The 

15 selected data stream or selected portions of data streams 
are displayed to user through output system 506, which 
may be used to drive displays and audio output devices, 
such as a high definition television or speakers. 
With reference now to Figure 6, a diagram 

20 illustrating an example display is depicted in accordance 
with a preferred embodiment of the present invention. In 
this example, a presentation system 600 is presented with 
a display unit 602, audio sources 604, 606, and 608. 
Presentation system 600 is the part of a client through 

25 which the user sees and hears programs. In this example, 
audio source 604 is a left speaker while audio source 608 
is a right speaker. Audio source 606 is the center 
speaker in this example. The video 610 is displayed in 
display 602 with text box 612, text box 614, and video 

30 616. Video 610 is the primary data stream in this 

example with text box 612 and text box 614 being overlaid 
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over video 610 . 

Text box 612 and text box 614 are generated from 
two data streams in this example. Text box 612 is a 
stationary text box, which may be used to provide 
5 information, such as statistics about the program, 

upcoming programs, or any other information the user may 
select. Text box 614 in this example is a ^^pop-up," 
which may be used to provide information about a 
particular object or person in video 610. 

10 Additionally, the data packets in text box 614 may 

include data to reposition text box 614 to follow a 
particular object or person. Video 616 is displayed 
using another data stream to provide additional viewing 
for the user. For example, video 616 may provide 

15 different perspectives in the program or may illustrate 
scene not currently displayed within video 610. 
Initially, the user may select various audio streams for 
feed through 604, 606, and 608. For example, if the 
programming being displayed is a sporting event, the user 

20 might select a particular player and have audio 

information about that player played over 608 or have 
direct feeds from a microphone located on the player 
played through 604. 

By way of example, consider a football game being 

25 broadcast across the Internet or a broadband network. 

The end user is able to select from and mix several audio 
streams. For example, the quarterback microphone of 
either team, the coach's microphone of either team, 
and/or the announcers, of which, there could be several 

30 (e.g., John Madden, Phil simms, Howie Long, Bruce Beadle, 
Michael Paolini, etc.). In this manner, the user is able 
to tune-out (deselect) John Madden if they find him 
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offensive. The user might instead choose to listen to 
the Quarterback microphone of the 49er's (or the opposing 
team, or another announcer of their choice, or to listen 
in another language entirely) without having to change 
5 the channel/coverage. 

Carrying the football example further, with the 
present invention, it is possible to have a separate 
audio track for viewers that are new to the sport. This 
separate audio track would explain the rules of the game 
10 and what is happening in additional detail. It would 

also explain the technical jargon particular to football. 
The user may choose to have this additional information 
presented as a text stream. 

In yet another example of the use of the present 
15 invention, continuing with the football theme, the user 
could place team A' s player voices on the right channels 
with team B' s player voices on the left channels and the 
announcers' voices on the rear channels. Furthermore, 
the user could select the player with the ball such that 
20 this player's voice is always on the center channel. 

One primary data stream representing the video 
(which does not change no matter which audio overlay 
streams are selected) , and many small add-on data streams 
representing audio or video overlays may be used to 
25 present the program. These streams could be mixed 
together or assigned to channels (right, left, rear, 
center, etc.) and/or 3-D positional locations using the 
latest "'Dolby Digital" from Dolby Laboratories and other 
similar technologies. In addition, the volume could be 
30 adjusted on a per-overlay basis. More information on 
technologies, such as ''Dolby Digital" may be found in 
United States Patent No. 5,912,976. 

With respect to video overlays, several video 
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overlays can also be streamed and selected in a manner 
similar to the audio streaming and mixed to ^'overlay'' the 
final video. If, for example, the desired overlay was a 
balloon shaped pop-up at a certain video display position 

5 containing some text, then the additional overlay stream 
would not have to be video itself, but rather the 
information to create the overlay video* Therefore, the 
size of the data stream would be much, much smaller than 
a primary video stream. Additionally, unrelated 

10 information in addition to the data streams selected for 
an event, such as, for example, stock market data or news 
headlines also may be provided to the user. If, for 
example, the event or programming being viewed is that of 
a football game, it is possible to have a video overlay 

15 stream dedicated to displaying pop-up balloons with data, 
such as statistics on players in which these pop-up 
balloons may appear over the players themselves. 
Additionally, betting odds, other interesting facts about 
the players, or rules and information about events may be 

20 displayed in this manner. Another type of data stream 

may include embedded pictures of players with biographies 
tagged on as well. 

For other types of programming, other pop-up windows 
may be provided. For example, for a murder mystery show, 

25 pop-up windows giving clues about future events may be 
displayed or facts about actors, the script, or director 
also may be displayed in association with the program. 
In the case of a murder mystery show, different levels of 
clues may be provided depending the selection made by the 

30 user. 

In this manner, in the context of the football game 
example, it is possible to have video overlay streams 
dedicated to displaying pop-up balloon (s) with data such 
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10 



as: statistics on players (they very well might appear 
over the players themselves), betting odds, odd or 
interesting facts, information explaining what is 
happening and rules (for novices) , a summary of different 
on-going sporting event data (such as scores and 
statistics from other games in progress), stock market 
data, and news headlines to provide but a few examples. 
Another stream type might contain embedded pictures of 
the players with bios tagged on as well. 

In another embodiment, it is possible to make the 
overlays selectable. That is to say that the user might 
do the equivalent of "clicking" on one, to trigger 
additional overlay streams, or lock it in place. For 
example, selecting a player's "name overlay" might for 
15 example bring up an overlay with that player's biography. 
In a one-way system, this could be done by having an 
overlay channel which continually cycles through all the 
players biographies, and having the client "select" the 
one of interest at the time of broadcast. 

Additionally, the user could select between the 
different options using their client and thus have a lot 
more control of the information they would like to see 
and hear during a broadcast. 

Turning now to Figures 7A-7D, diagrams illustrating 
data structures contained within data streams sent to a 
client are depicted in accordance with a preferred 
embodiment of the present invention. In Figure 7A, a 
data packet 700 is shown in which the data packet 
includes a header 702 and a payload 704. Header 702 
contains information used in routing and handling data 
contained within payload 704 of data packet 700. 

In Figure 7B, an example of header information found 
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within a data packet, such as data packet 700, is shown. 
In this example, header 706 includes an ID field 708, a 
channel field 710, a position field 712, and a volume 
field 714. 

5 In this example, header 706 is a header identifying 

an audio packet. ID field 708 is used to identify the 
type of data packet. Channel field 710 in this example 
may be used to identify the channel or channels in which 
the data packet is to be presented to the user. For 

10 example, channel field 710 may be used to identify 
channels such as those in a surround system. The 
channels may be, in this example, right, left, rear, or 
center. Additionally, position field 712 also is 
included such that three dimensional positional locations 

15 using sound technologies such as Dolby Digital may be 
employed. This positional information may provide a 
default positioning that may be altered by a user. Volume 
field 714 is included to allow for adjusting the volume 
of the data in this data packet relative to other data 

20 packets to allow for overlays or emphasis for different 
data streams. Other fields may be used in addition to or 
in place of fields shown in header 706 for presenting 
audio information . 

In Figure 7C, a header 716 is an example of header 

25 information found in a video packet used in a video 

stream. In this example, header 716 includes an ID field 
718, a time stamp field 720, and a CRC field 722. ID 
field 718 is used to identify the type of data contained 
within the packet, while time stamp 720 in this example 

30 is used for synchronization purposes. CRC field 722 may 
be used for error checking or synchronization purposes 
depending upon the implementation. If CRC field 722 is 
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used for synchronization purposes^ the CRC data is 
calculated for another data stream and placed within this 
field prior to being transmitted or broadcast to a 
viewer. The CRC data may be^ for example, calculated for 
5 a frame or a portion of a frame in the other data stream. 
When this data stream and the other data stream are 
received, CRC data is calculated for a frame in the other 
data stream. This calculated CRC data is compared with 
the CRC data stored in CRC field 722. Matches between 

10 these CRC values are used to identify synchronization 
points between the two data streams • 

Next, in Figure 7D, another example of a header used 
for data packets in a video stream is illustrated. 
Header 724 includes an ID field 726, a type field 728, a 

15 location field 730, a duration field 732, and a z depth 
field 734. ID field 726 is used to identify the data 
packet as containing video data. Type field 728 in this 
example may be used to identify the type of video, such 
as text, or a frame in a moving video stream. Location 

20 field 730 may be used to identify the location on the 
screen at which the data is to be displayed. Duration 
field 732 may identify the amount of time the data is to 
be displayed. Z depth field 734 may be used to identify 
the depth or order in which the data is to be displayed 

25 relative to other video streams. 

In the example, header 724 is an example of a header 
used for a video stream containing text to be displayed 
as a balloon or pop-up on the screen. In this case, 
duration field 732 identifies how long the text is to be 

30 displayed while the z depth field 734 identifies where 
the text is to be located with respect to other video 
streams. The z depth is used to determine the order in 
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which video streams or overlays are displayed on the 
display. Location field 730 provides the xy location on 
the screen. In the text for display in the pop-up would 
be found in the payload. In this manner, a video overlay 
5 stream dedicated to displaying text, may be used along 
with moving video data streams. 

The present invention also provides for 
selectability of these different data streams. For 
example, user may be able to press a selected button to 

10 trigger additional overlay streams or to lock one in 

place. For example, selecting a player's "name overlay" 
might bring up an overlay with that player's biography. 
In a one way system, this may be performed by having a 
overlay channel which continually cycles through all of 

15 the players' biographies and having the client machine 
select the one of interest at the time of broadcast. In 
this manner, user may select between different options 
and have more control on the information that the user 
desires to see and hear during a program. The selections 

20 may be implemented via a one way or two way communication 
depending on the bandwidth and type of network in use. 

Overlay streams may be controlled from the server. 
In such a case, a command may be imbedded in an overlay 
or a set of commands may be imbedded to cause overlays to 

25 be selected or deselected on the client. Such a 

mechanism would allow for switching selected overlays off 
to broadcast an important event, and then resuming the 
previous programming. 

Turning now to Figure 8, a flowchart of a process 

30 used to process data streams is depicted in accordance 

with a preferred embodiment of the present invention. To 
start, a user selects programming to be retrieved from a 
distributed database (step 800). The user receives 
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multiple video and audio streams corresponding to the 
selected program (step 802) . The user selects audio 
streams (step 804), either directly in real time or 
through the use of user defined configurations, and; if 
5 more than one audio stream is selected (step 806) , then 
the user configures the presentation profile for the 
selected audio streams (step 808) . For example, if three 
audio streams are chosen, the user could configure the 
system to play one audio stream on the left channel, one 

10 audio stream on a center channel, and one audio stream on 
a right channel. Alternatively, the user could configure 
the presentation of the selected audio streams such that 
the volume of one audio stream is louder than the volume 
of the other two audio streams. 

15 Next, the user selects, from the plurality of video 

streams received, one or more video streams to be 
displayed (step 811) . If more than one video stream is 
selected (step 814), then the user configures the 
presentation of the multiple video streams (step 816) , 

20 For example, one video stream could be presented on the 
left side of the video display and a second video stream 
could be presented on the right side of the video 
display. In another example, one video stream might 
occupy the entire video display screen, while a second 

25 video stream was placed in a smaller window within the 

first video stream. Also, video streams are configurable 
for such things as opacity and scaling, as well as other 
factors. Opacity is the semi-transparency as seen in 
menus overlaid on an event in which the event can be seen 

30 through the menu. Scaling may be employed to size 
display of a video stream on a display device. 

Once the media streams are selected and configured 



23 

Docket No. AT9-99-254 

by the user, the audio and video streams are synchronized 
to each other (step 818) and presented to the user in a 
user configured manner (step 820) . 

With reference again to step 806, if more than one 
5 audio stream is not selected, then the user selects, from 
the plurality of video streams received, one or more 
video streams to be displayed (step 811) • 

With reference again to step 814, if more than one 
video stream is not selected, the audio and video streams 
10 are synchronized to each other (step 818) and presented 
to the user in a user configured manner (step 820) . 
Several mechanisms may be employed to synchronize the 
data streams. For example, a time stamp may be sent with 
each packet of data in each media stream, then each 
15 selected data stream is synchronized to be presented to 
the user such that data packets with identical time 
stamps are presented at the same time. 

Each data stream also may contain a spike in the 
data that appears at a periodic rate. The spike in the 
20 data may be, for example, a data packet that contains no 
video or audio. Further, each data stream contains a 
larger spike that appears at a longer periodic rate. For 
example, the shorter spike may appear every 30 frames 
wherein the larger spike appears every 10 seconds. The 
25 selected data streams may be synchronized such that they 
are presented to the user such that the spikes appear in 
the data streams simultaneously. The longer spike 
enables the system to resynchronize itself periodically 
if the data streams have become separated by a period of 
30 one or two smaller data spikes. 

In a third method of synchronization, cyclic 
redundancy check (CRC) values. Synchronizing data streams 
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using a CRC data involves calculating CRC data for a 
first data stream and placing those calculated values in 
the headers of data packets in the second data stream. 
The CRC values are placed in data packets in the second 
5 data stream that should be synchronized with data packets 
in the first data stream. 

Turning to Figure 9, a flowchart of a process for 
generating CRC values for use in synchronizing data 
streams is depicted in accordance with a preferred 

10 embodiment of the present invention. The process 

illustrated is used for video streams, but may be applied 
to other types of data streams. 

The process begins by selecting an unprocessed frame 
(step 900) . This step selects the first unprocessed 

15 frame for content that will be transmitted as a data 
stream. A CRC value is then calculated for the frame 
(step 902) . The CRC value may be calculated for an 
entire frame or different portions of the frame. For 
example, the CRC value may be based on the first five 

20 bytes of the frame, five bytes in the middle of the 
frame, and the last five bytes in the frame. 

Next, the portion of a second data stream that is to 
be synchronized with this frame is identified (step 904). 
The CRC value is then placed into the header of a data 

25 packet for that portion of the data stream (step 906) . A 
determination is then made as to whether additional 
unprocessed frames are present for processing (step 908) . 
If additional unprocessed frames are present, the process 
returns to step 900. Otherwise, the process terminates. 

30 The process in Figure 9 may be used when the data 

streams are transmitted or broadcast. Alternatively, 
these processes may be initiated prior to transmitting or 
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broadcasting a data stream. In such a case, the CRC 
values may be stored and added to the second data stream 
when that data stream is transmitted. 

Turning now to Figure 10, a flowchart of a process 
5 for synchronizing data streams using CRC data is depicted 
in accordance with a preferred embodiment of the present 
invention. The processes illustrated in Figure 9 may be 
implemented at the viewer site to synchronize incoming 
data streams for presentation of a program or event. 

10 The process begins by receiving data streams (step 

1000) . In step 1000, the data streams are received for 
presentation. These data streams are placed in a buffer 
prior to presentation (step 1002) , A CRC value is 
calculated for a frame in the first data stream stored in 

15 the buffer (step 1004) . Thereafter, the calculated CRC 
value is compared to CRC values in the second data stream 
stored in the buffer to identify a point in the second 
data stream containing a matching CRC value (step 1006) . 
In the depicted examples, CRC values are placed within 

20 headers in the second data stream at points where the CRC 
value should match a calculated CRC value from a portion 
of the first data stream. Thereafter, the first and 
second data stream are presented at the matching or 
synchronization points (step 1008) with the process then 

25 returning to step 1000. The present invention could be 
implemented via one-way or two-way communication 
depending on the bandwidth of the network type in use. 

Thus, the present invention provides a customized 
experience for a user. The present invention provides 

30 this customized experience by providing a number of 

different types of data streams, audio and video, which 
are delivered to a client in which the client may select 
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from these data streams to customize the program or event 
being viewed by the user. Further, the present invention 
allows for decreasing the amount of bandwidth needed to 
provide customized programming. Currently, a data stream 
5 is created for each version or customization of a program 
transmitted to viewers in which the audio and video 
cannot be selectively modified. This advantage is 
provided by sending a set of data streams from which 
different users may select a subset of the data streams 

10 for customizing a program or event, rather than creating 
a new data stream in which audio and video are 
transmitted in an unmodifiable form for each desired 
customization of a program or event. 

It is important to note that while the present 

15 invention has been described in the context of a fully 
functioning data processing system, those of ordinary 
skill in the art will appreciate that the processes of 
the present invention are capable of being distributed in 
the form of a computer readable medium of instructions 

20 and a variety of forms and that the present invention 
applies equally regardless of the particular type of 
signal bearing media actually used to carry out the 
distribution. Examples of computer readable media 
include recordable-type media such a floppy disc, a hard 

25 disk drive, a RAM, and CD-ROMs and transmission-type 
media such as digital and analog communications links. 

The description of the present invention has been 
presented for purposes of illustration and description, 
but is not intended to be exhaustive or limited to the 

30 invention in the form disclosed. Many modifications and 
variations will be apparent to those of ordinary skill in 
the art. The embodiment was chosen and described in 
order to best explain the principles of the invention, 
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the practical application, and to enable others of 
ordinary skill in the art to understand the invention for 
various eitibodiments with various modifications as are 
suited to the particular use contemplated. 
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CLAIMS 

What is claimed is: 

1. A method in a data processing system for user 
5 controlled selection of multimedia data streams for an 
event, the method comprising: 

receiving a set of video streams; 

receiving a set of audio streams; 

presenting selected ones of the set of video 

10 streams; 

presenting selected ones of the set of audio 

streams; and 

responsive to user input to the data processing 
system, selectively altering the selected ones of the 
15 set of video streams and the selected ones of the set of 
audio streams presented for the event. 

2. The method of claim 1, wherein the video stream is 
presented on a display and wherein the step of 

20 selectively altering the selected ones of the set of 

video streams and the portion of the set of audio streams 
presented includes: 

altering a location in the display in which the 
video stream is presented. 

25 

3. The method of claim 1, wherein the step of 
selectively altering the selected ones of the set of 
video streams and the portion of the set of audio streams 
presented includes: 

30 selecting different selected ones of the set of 

video streams presentation. 



4. The method of claim 1, wherein the step of 



29 

Docket No. AT9-99-254 

selectively altering the selected ones of the set of 
video streams and the selected ones of the set of audio 
streams presented includes: 

selecting additional selected ones of the set of 
5 video streams for presentation. 

5. The method of claim 1, wherein the step of 
selectively altering the selected ones of the set of 
video streams and the selected ones of the set of audio 

10 streams presented includes; 

selecting different selected ones of the set of 
audio streams presentation. 

6. The method of claim 1, wherein the step of 

15 selectively altering the seleced ones of the set of video 
streams and the selected ones of the set of audio streams 
presented includes; 

selecting another selected ones of the set of 
audio streams for presentation, 

20 

7. The method of claim 1 further comprising: 
receiving a set of information streams including 

text; and 

responsive to user input, selectively presenting 
25 selected ones of the set of information streams on a 
display. 

8. The method of claim 1, wherein the set of video 
streams and the set of audio streams include time stamps 

30 and further comprising: 

synchronizing the selected ones of the video stream 
with the selected ones of the audio stream using the time 
stamps . 
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9. The method of claim 1^ wherein the set of video 
streams and the set of audio streams include data packets 
located in the video and audio data streams periodically 
5 and further comprising: 

synchronizing the selected ones of the video stream 
with the selected ones of the audio stream using the data 
packets . 

10 10. The method of claim 1, wherein some selected streams 
from the set of video streams and the set of audio 
streams include reference cyclic redundancy check data 
based on other streams from the set of video streams and 
the set of audio streams and further comprising: 

15 synchronizing a portion of a first stream from the 

selected streams with a portion of a second stream from 
the other streams by calculating calculated cyclic 
redundancy check data for the second stream and comparing 
the calculated cyclic redundancy check data with the 

20 reference cyclic redundancy check data from the first 
stream to identify synchronization points. 

11. The method of claim 1, wherein the data processing 
system is a computer. 

25 

12. The method of claim 1, wherein the data processing 
system is a personal digital assistant, 

13. The method of claim 1^ wherein the data processing 
30 system is a television. 

14. A method for tailoring a multimedia presentation of 
an event on a computerized multimedia system comprising 
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the steps of: 

providing a set of video streams, a set of audio 
streams and a set of information streams for the event 
via a network coupled to the computerized multimedia 
5 systems- 
selecting video streams for presentation from the 
set of video streams for the event; 

selecting audio streams for presentation from the 
set of audio streams for the event; 
10 selecting information streams for presentation from 

the set of information streams for the event; 

responsive to user input, assigning selected video 
streams and the selected audio streams to respective 
portions of video and audio output devices; and 
15 presenting the event according to selected video 

stream and audio stream assignments. 

15. The method of claim 14, wherein the step of 
selecting video streams for presentation from the set of 

20 video streams for the event is performed in the 
computerized multimedia system^ 

16. The method of claim 14, wherein the step of 
selecting audio streams for presentation from the set of 

25 audio streams for the event is performed in the 
computerized multimedia system. 

17. The method as recited in claim 14, wherein the set 
of video streams and the set of audio streams are 

30 provided from a first source. 

18. The method as recited in claim 11, further 
comprising: 
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responsive to user selection, providing a second 
video stream from a second source, 

19. The method as recited in claim 17, further 
5 comprising: 

responsive to user selection, providing a second 
audio stream from a second source. 

20. The method as recited in claim 14, wherein the set 
10 of video streams, the set of audio streams, and the set 

of information streams are provided from at least two 
different sources. 

21. The method as recited in claim 14, wherein the set 
15 of video streams, the set of audio streams, and the set 

of information streams is provided via a broadband 
network. 



22. A data processing system for user controlled 
20 selection of multimedia data streams for an event, the 
data processing system comprising: 

first receiving means for receiving a set of video 
streams; 

second receiving means for receiving a set of audio 
25 streams; 

first presenting means for presenting selected ones 
of the set of video streams; 

second presenting means for presenting selected ones 
of the set of audio streams; and 
30 altering means, responsive to user input to the data 

processing system, for selectively altering the selected 
ones of the set of video streams and the selected ones of 
the set of audio streams presented for the event. 
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23. The data processing system of claim 22, wherein the 
video stream is presented on a display and wherein the 
altering means includes: 

5 altering means for altering a location in the 

display in which the video stream is presented, 

24. The data processing system of claim 22, wherein the 
altering means: 

10 first selecting means for selecting different 

selected ones of the set of video streams presentation. 

25. The data processing system of claim 22, wherein the 
altering means includes: 

15 second selecting means for selecting additional 

selected ones of the set of video streams for 
presentation. 

26. The data processing system of claim 22, wherein the 
20 altering means includes: 

third selecting means for selecting different 
selected ones of the set of audio streams presentation. 

27. The data processing system of claim 22, wherein the 
25 altering means includes: 

fourth selecting means for selecting another 
selected ones of the set of audio streams presentation. 

28. The data processing system of claim 22 further 
30 comprising: 

receiving means for receiving a set of information 
streams including text; and 

presenting means, responsive to user input, 
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selectively for presenting selected ones of the set of 
information streams on a display. 

29. The data processing system of claim 22, wherein the 

5 set of video streams and the set of audio streams include 
time stamps and further comprising: 

first synchronizing means for synchronizing the 
selected ones of the video stream with the selected ones 
of the audio stream using the time stamps. 

10 

30. The data processing system of claim 22, wherein the 
set of video streams and the set of audio streams include 
data packets located in the video and audio data streams 
periodically and further comprising: 

15 second synchronizing means for synchronizing the 

selected ones of the video stream with the selected ones 
of the audio stream using the data packets. 

31. The data processing system of claim 22, wherein some 
20 selected streams from the set of video streams and the 

set of audio streams include reference cyclic redundancy 
check data based on other streams from the set of video 
streams and the set of audio streams and further 
comprising: 

25 third synchronizing means for synchronizing a 

portion of a first stream from the selected streams with 
a portion of a second stream from the other streams by 
calculating calculated cyclic redundancy check data for 
the second stream and comparing the calculated cyclic 

30 redundancy check data with the reference cyclic 

redundancy check data from the first stream to identify 
synchronization points. 
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32. The data processing system of claim 22, wherein the 
data processing system is a computer. 

33. The data processing system of claim 22, wherein the 
5 data processing system is a personal digital assistant, 

34. The data processing system of claim 22, wherein the 
data processing system is a television, 

10 35. A data processing system for tailoring a multimedia 
presentation of an event on a computerized multimedia 
system, the data processing system comprising: 

providing means for providing a set of video 
streams, a set of audio streams and a set of information 

15 streams for the event via a network coupled to the 
computerized multimedia system; 

first selecting means for selecting video streams 
for presentation from the set of video streams for the 
event; 

20 second selecting means for selecting audio streams 

for presentation from the set of audio streams for the 
event; 

third selecting means for selecting information 
streams for presentation from the set of information 
25 streams for the event; 

assigning means, responsive to user input, for 
assigning selected video streams and the selected audio 
streams to respective portions of video and audio output 
devices; and 

30 presenting means for presenting the event according 

to selected video stream and audio stream assignments. 

36. The data processing system of claim 35, wherein the 
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first selecting means includes selecting video streams 
for presentation from the set of video streams for the 
event is performed in the computerized multimedia system. 

5 37. The data processing system of claim 35, wherein the 
second selecting means for selecting audio streams for 
presentation from the set of audio streams for the event 
is performed in the computerized multimedia system. 

10 38. The data processing system as recited in claim 35, 
wherein the set of video streams and the set of audio 
streams are provided from a first source. 

39. The data processing system as recited in claim 38, 
15 further comprising, responsive to user selection, 

providing a second video stream from a second source. 

40. The data processing system as recited in claim 38, 
further comprising : 

20 providing means, responsive to user selection, for 

providing a second audio stream from a second source. 

41. The data processing system as recited in claim 35, 
wherein the set of video streams, the set of audio 

25 streams, and the set of information streams are provided 
from at least two different sources. 

42. The data processing system as recited in claim 35, 
wherein the set of video streams, the set of audio 

30 streams, and the set of information streams is provided 
via a broadband network. 

43. A computer program product in a computer readable 
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medium for user controlled selection of multimedia data 
streams for an event, the computer program product 
comprising: 

first instructions for receiving a set of video 
5 streams; 

second instructions for receiving a set of audio 
streams; 

third instructions for presenting selected ones of 
the set of video streams; 
10 fourth instructions for presenting selected ones of 

the set of audio streams; and 

fifth instructions, responsive to user input to the 
data processing system, for selectively altering the 
selected ones of the set of video streams and the 
15 selected ones of the set of audio streams presented. 



44. A computer program product in a computer readable 
medium for tailoring a multimedia presentation of an 
event on a computerized multimedia system comprising: 

20 first instructions for providing a set of video, 

audio and information streams for the event via a network 
coupled to the computerized multimedia system; 

second instructions for selecting video streams for 
presentation from the set of available video streams for 

25 the event; 

third instructions for selecting audio streams for 
presentation from the set of available audio streams for 
the event; 

fourth instructions for selecting information 
30 streams for presentation from the set of available 
information streams for the event; 

fifth instructions, responsive to user input, for 
assigning the selected video and audio streams to 
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respective portions of video and audio output device 
and 

sixth instructions for presenting the event 
according to the selected video stream assignments. 
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ABSTRACT OF THE DISCLOSURE 

5 METHOD AND APPARATUS FOR USER-CONTROLLED SELECTIVE 

OVERLAY IN A STREAMING MEDIA 

A method and system for tailoring a multimedia 
presentation of an event on a computerized multimedia 

10 system to meet a user's desires. In a preferred 

embodiment, a set of video streams and a set of audio 
streams for the event are provided to the user via a 
network coupled to the multimedia system. From the set 
of available video streams for the event, one or more 

15 video streams are selected for presentation to the user. 
From the set of available audio streams for the event, 
one or more audio streams are selected for presentation 
to the user. Furthermore, the relative volumes of the 
different audio streams may be adjusted. In response to 

20 user input, the selected video and audio streams are 

assigned to respective portions of video and audio output 
devices. The event is presented to the user according to 
the selected video stream assignments. 
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James H. Barksdale, Jr., Reg. No. 24,091; Casimer K. Salys, Reg. No. 28,900 
Robert M. Carwell, Reg. No. 28,499; Douglas H. Lefeve, Reg. No. 26,193 
Jeffrey S. LaBaw, Reg. No. 31,633; David A. Mims, Jr., Reg. 32,708; Volei 
Emile, Reg. No. 39, 969; Anthony V. England, Reg. No. 35,129; Leslie A. Van 
Leeuwen, Reg. No. 42,196; Christopher A. Hughes, Reg. No. 26,914; Edward A. 
Pennington, Reg. No. 32,588; John E. Hoel, Reg, No. 26,279; Joseph C. Redmond, 
Jr., Reg. No. 18,753; Marilyn S. Dawkins, Reg, No. 31,140; Duke W. Yee, Reg, 
No. 34,285; Mark E. McBurney, Reg. No. 33,114; and Colin P. Cahoon, Reg, No, 
38,836; Joseph R. Burwell, Reg, No. 44,468; Rudolph J. Buchel, Reg, No. 
43,448; Stephen R. Loe, Reg. No. 43,757. 

Send correspondence to: Duke w. Yee, Carstens, Yee & Cahoon, LLP, P.O. Box 
802334, Dallas, Texas 75380 and direct all telephone calls to Duke W. Yee, 
(972) 362-2001 

FULL NAME OF SOLE OR Fr^S^ INVENTOR: RONALD W. BASSBTT 

INVENTORS SIGNATURE: 0<CJJ /S^^^^M-- bate: 

RESIDENCE: 1002 HACKBERRY DRIVE 

PFLUGERVILLE> TEXAS 78660 
CITIZENSHIP: UNITED STATES 
POST OFFICE ADDRESS: SAME AS ABOVE 



FULL NAME OF SECOND INVENTOR: BRUCE A. BEADLE 

INVENTORS SIGNATURE: 7""^-^^ /U ^'^P^^.^-^.^M. ^ DATE : " 2 O — ^ 9 

RES I DENCE : 3648 FLORA VISTA LOOP 

ROUND ROCK, TEXAS 78681 

CITIZENSHIP: UNITED STATES 

POST OFFICE ADDRESS: SAME AS ABOVE 
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DOCKET NUMBER: AT9-99 -254 



FULL NAME OF THIRD INVENTOR; MICHAEL WAYNE BROWN 

INVENTORS SIGNATURE x ^J^^^Adj^^,^^^^ DATE: 

RES I DENCE : 529 RIVER DOWN ROAD 

GEORGETOWN, TEXAS 78268 

CITIZENSHIP: UNITED STATES 

POST OFFICE ADDRESS: SAME AS ABOVE 



FULL NAME OF FOURTH INVENTOR: LEON fir^OUD 



INVENTORS SIGNATURE: 





4m- 



RESIDENCE: 1 IjQ I-,iiQBBrY--HORSE-TmR^ 
^ 

CITIZENSHIP: UNITED STATES 

POST OFFICE ADDRESS: SAME AS ABOVE 



DATE 




FULL NAME OF FIFTH INVENTS": MICHAEL A, PAOLINI 
INVENTORS SIGNATURE :_lMLz_i_ 



RESIDENCE: 1406 TERRA STREET 

ROUND ROCK, TEXAS 78664 

CITIZENSHIP: UNITED STATES 

POST OFFICE ADDRESS: SAME AS ABOVE 



DATE: 
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